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DECISION ON APPEAL 
This is a decision on appeal under 35 U.S.C. § 134 from the 
examiner's final rejection of claims 1-9, 11, 12, 14, 16-22, 24, 
33-40, 42, 44, 46-51, 53, 54, 56 and 58-63. Claims 10, 13, 15, 
23, 25-32, 41, 43, 45, 52, 55 and 57 have been objected to by the 
examiner . 
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BACKGROUND 



Appellants' invention relates to data structure for 
providing a user interface to objects, wherein the user interface 
exploits spatial memory and visually indicates at least one 
object parameter. An understanding of the invention can be 
derived from a reading of exemplary claim 1, which is reproduced 
as follows: 

1. A man-machine interface method for permitting a user to 
act on thumbnails, each thumbnail representing an associated 
object containing information, for use with a machine having a 
video display device and a user input device, the man-machine 
interface method comprising: 

a) generating a three-dimensional environment, having a 
depth, to be rendered on the video display device; 

b) determining a two-dimensional location and a depth of 
each of the thumbnails in the three-dimensional environment, 
wherein, for each of the thumbnails, the depth is a function of 
at least one parameter of the object associated with the 

t humbna i 1 ; and 

c) generating the thumbnails within the three-dimensional 
environment, at the determined two-dimensional locations and 
depths, to be rendered on the video display device. 

The prior art references of record relied upon by the 

examiner in rejecting the appealed claims are: 

Joskowicz et al . 5,669,006 Sep. 16, 1997 

( Joskowicz ) 

Baldwin 5,701,444 Dec. 23, 1997 



Robertson et al . 
(Robertson) 



6, 166,738 



Dec. 26, 2000 
(filed Sept. 14, 1998) 
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Claims 1-3, 5-9, 11, 12, 14, 16-22, 24, 33-40, 42, 44, 46- 
51, 53, 54, 56 and 58-63 stand rejected under 35 U.S.C. § 103(a) 
as being unpatentable over Robertson and Joskowicz. 

Claim 4 stands rejected under 35 U.S.C. § 103(a) as being 
unpatentable over Robertson, Joskowicz and Baldwin. 

Rather than reiterate the conflicting viewpoints advanced by 
the examiner and appellants regarding the above-noted rejections, 
we make reference to the examiner's answer (Paper No. 18, mailed 
August 26, 2003) for the examiner's complete reasoning in support 
of the rejections, and to appellants 1 brief (Paper No. 17, filed 
June 12, 2003) and reply brief (Paper No. 22, filed January 27, 
2004) for appellants' arguments thereagainst . Only those 
arguments actually made by appellants have been considered in 
this decision. Arguments which appellants could have made but 
chose not to make in the brief have not been considered. See 37 
CFR § 41.37 (c) (1) (vii) . 

OPINION 

In reaching our decision in this appeal, we have carefully 
considered the subject matter on appeal, the rejections advanced 
by the examiner, and the evidence of obviousness relied upon by 



Appeal No. 2004-2005 
Application No. 09/335,640 



Page 4 



the examiner as support for the rejections. We have, likewise, 
reviewed and taken into consideration, in reaching our decision, 
appellants' arguments set forth in the briefs along with the 
examiner's rationale in support of the rejections and arguments 
in rebuttal set forth in the examiner's answer. 

Upon consideration of the record before us, we reverse, 
essentially for the reasons set forth by appellants, and add the 
following comments . 

We begin with the rejection of claims 1-3, 5-9, 11, 12, 14, 
16-22, 24, 33-40, 42, 44, 46-51, 53, 54, 56 and 58-63 under 35 
U.S.C. § 103(a) as being unpatentable over the teachings of 
Robertson and Joskowicz. We turn first to claim 1. In rejecting 
claims under 35 U.S.C. § 103, it is incumbent upon the examiner 
to establish a factual basis to support the legal conclusion of 
obviousness. See In re Fine , 837 F.2d 1071, 1073, 5 USPQ2d 1596, 
1598 (Fed. Cir. 1988) . In so doing, the examiner is expected to 
make the factual determinations set forth in Graham v. John Deere 
Co^, 383 U.S. 1, 17, 148 USPQ 459, 467 (1966), and to provide a 
reason why one having ordinary skill in the pertinent art would 
have been led to modify the prior art or to combine prior art 
references to arrive at the claimed invention. Such reason must 
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stem from some teaching, suggestion or implication in the prior 
art as a whole or knowledge generally available to one having 
ordinary skill in the art. Uniroyal, Inc. v. Rudkin-Wiley Corp. , 
837 F.2d 1044, 1051, 5 USPQ2d 1434, 1438 (Fed. Cir. 1988); 
Ashland Oil, Inc. v. Delta Resins & Refractories, Inc. r 776 F.2d 
281, 293, 227 USPQ 657, 664 (Fed. Cir. 1985); ACS Hosp. Sys. . 
Inc. v. Montefiore Hosp. , 732 F.2d 1572, 1577, 221 USPQ 929, 933 
(Fed. Cir. 1984). These showings by the examiner are an 
essential part of complying with the burden of presenting a prima 
facie case of obviousness. Note In re Oetiker , 977 F.2d 1443, 
1445, 24 USPQ2d 1443, 1444 (Fed. Cir. 1992). If that burden is 
met, the burden then shifts to the applicant to overcome the 
prima facie case with argument and/or evidence. Obviousness is 
then determined on the basis of the evidence as a whole. See 
id^; In re Hedges , 783 F.2d 1038, 1039, 228 USPQ 685, 686 (Fed. 
Cir. 1986); In re Piasecki . 745 F.2d 1468, 1472, 223 USPQ 785, 
788 (Fed. Cir. 1984); and In re Rinehart , 531 F.2d 1048, 1052, 
189 USPQ 143, 147 (CCPA 1976). 

The examiner's position (answer, page 5) is that M 0]hile 
Robertson patent discloses a perspective view facility 252 to 
determine the thumbnail object location or position information 
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(col. 17, lines 29-61), the location or position information of 
Robertson's does not explicitly show further information about 
the object, such as the depth parameter of the object thumbnail 
on the three dimensional surface is not explicitly shown." To 
overcome this deficiency of Robertson, the examiner turns to 
Joskowicz for a teaching of a relationship value (parameter) 
between episodes, wherein each episode is represented as a 
thumbnail . 

Appellants assert (brief, page 5) that claim 1 recites, in 
part, that "for each of the thumbnails, the depth is a function 
of at least one parameter of the object associated with the 
thumbnail" (bold italics and underlining omitted) . Appellants 
assert that neither reference, alone or in combination, teach or 
suggest this limitation, because the cited references do not 
associate depth of a rendered thumbnail with at least one object 
associated with the thumbnail. It is argued ( id. ) that Joskowicz 
simply teaches rendering of icons (including Z-axis based 
rendering) as a function of the icons themselves (e.g., size, 
area, predefined constraints relating to spacing of the icons) 
but not as a function of at least one parameter of an object 
associated with the displayed thumbnail as defined in the claimed 
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invention. Appellants assert (answer, page 6) that Joskowicz 
merely suggests Z-ordering thumbnails by surface area so that 
larger thumbnails do not obscure smaller thumbnails; i.e., 
Joskowicz teaches rendering an icon as a parameter associated 
directly with the icon (the depth of an icon is directly related 
to the physical size of the icon (height times width) . 
Appellants further assert ( id. ) that both Robertson and Joskowicz 
are concerned with rendering of thumbnails /icons so as to 
optimize utilization of valuable screen real estate via three- 
dimensional rendering of the thumbnails, whereas neither 
reference teaches or suggests that the placement of the 
thumbnails in three-dimensional space is also a function of at 
least one parameter of an object associated with the rendered 
thumbnail . 

The examiner responds (answer, pages 5 and 6) it would have 
been obvious to incorporate the depth information of Joskowicz 
with Robertson's three-dimensional environment "because the 
incorporation will enable Robertson to display or edit the values 
of the thumbnail associated with the displayed object (such as 
the document shown in Figs 8A-18) . Thus, the combination of the 
Robertson reference and Joskowicz reference would result in the 
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invention recited in claims 1 and 47." The examiner additionally 
argues (answer, page 16) that "Joskowicz also teaches depth 
associated with parameter such as priority or arrangement of an 
object (episode) associated with thumbnail (fig.l)." Moreover, 
the examiner asserts (answer, page 17) that Joskowicz defines the 
Z-ordering of episodes according to their relationship, or 
according to their arrangement or priority in the Z-ordering 
list, and adds that n [t]hus, combining the ( x 738) with ( x 006) 
results in determining the depth as a function of parameter of 
the object associated with the thumbnail." The examiner further 
asserts (answer, page 18) that Mi]n contrast to Appellant [sic] 
argument [ , ] Robertson and Joskowicz teach the depth as a 
function of parameter (such as priority or arrangement of 
episodes) of the object associated with the thumbnail)." 

In the reply brief, appellants add (page 6) that "Joskowicz 
et al . merely suggests 'z-ordering' thumbnails by surface areas 
so that larger thumbnails do not obscure smaller thumbnails", and 
( id. ) w teaches rendering an icon as a function of a parameter 
associated directly with the icon, i.e. the depth of the icon is 
directly related to the physical size of the icon (height times 
width) " (bold highlighting omitted) . 
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From our review of Joskowicz, we find that Joskowicz is 
drawn to a method for automatically obtaining spatial layout for 
multimedia presentations, and more particularly (col. 1, lines 
8-10) to automatically generating a spatial layout of the visible 
segments on a computer display screen. Joskowicz discloses (col. 
1, lines 13-17) that multimedia documents such as encyclopedias 
have become popular since this type of media is capable of 
storing a large amount of data, such as text, graphics, action 
video, sound, etc. Joskowicz explains that the user of a 
multimedia document typically presents or receives multimedia 
information called fragments, segments or objects, which 
Joskowicz hereinafter refers to as episodes. These episodes 
include audio and video clips, musical recordings, speech, typed 
text, still pictures, animation, etc. (col. 1, lines 18-26). 
Multimedia authoring is a process of ordering multimedia objects 
such as texts, graphics, sounds and videos in time and space. A 
set of multimedia objects or episodes, which have been ordered, 
is referred to as a story. Ordering the objects with respect to 
a story is done in a temporal dimension and in the spatial 
dimension. The invention is related to the problem of obtaining 
a spatial layout (col. 1, lines 27-34). In addition, a set of 
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objects in a multimedia story that appear simultaneously is 
called a clique (col. 1, lines 37 and 38) . With a large number 
of objects having a complicated spatial relationship, it becomes 
an involved and tedious process to find a suitable spatial design 
(col. 1, lines 41-44) . 

According to the invention, a spatial design for a 
multimedia clique is defined in three dimensions, where the third 
dimension defines the depth of overlapping objects (col. 1, lines 
58-62). The locations of the associated episodes and their 
respective depths on the screen are found. This process involves 
resolving space requirements between the several episodes in the 
clique according to a defined set of constraints (col. 1, lines 
62-66) . If a resolution can be reached which provides a 
consistent layout, the layout is displayed. The user is then 
given the opportunity to change the predefined set of 
constraints, resulting in the display of the layout with the 
changed set of constraints. If the layout is still not 
satisfactory, the user can change the sizes and locations of the 
individual episodes in the clique (col. 1, line 66 through col. 
2, line 6) . When the spatial design is finally accepted by the 
user, the process outputs the spatial design for the clique (col. 
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2, lines 9 and 10) . Given a series of cliques that constitute an 
entire multimedia story, we use a technique of constraint 
propagation, where an object in one clique may be displayed with 
other objects in a succeeding clique. Rather than moving that 
object around, the option is provided to fix the position of the 
object and let the other objects move around the fixed one (col, 
2, lines 11-17) . 

In practicing the invention, multimedia objects in a story 
are automatically arranged in a spatial layout, but not all 
objects are visible because an audio episode or time delay have 
no screen real estate. These invisible objects are ignored, and 
only visible objects are considered in the spatial design of the 
multimedia presentation. For visual episodes, the location and 
space they occupy on the screen may vary. In the case where 
there are overlaps between objects on the display screen, finding 
a spatial design for a multimedia clique can be characterized as 
a three dimensional problem in XYZ space, where Z is the depth of 
overlapping objects (col, 3, lines 21-36). To obtain a screen 
layout for a clique, we find the locations of the episodes and 
their depth. Let E denote the episodes in a multimedia clique C. 
Assume that each of the visual episodes occupy a rectangular 
region on the screen, and that each episode is associated with an 
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optimal area and aspect ratio. The area is defined by its width 
and height, and the aspect ratio is the ratio of its width to its 
height. Assume that for each episode, a set of constraints such 
as minimum area and prespecified range of aspect ratios, are 
given. Furthermore, relative placement constraints may also be 
given for the episodes. An example of this is illustrated in 
figure 2, below, 




F1G.2 



where a plurality of episodes comprising a clique are shown, with 
some episodes overlapping (col. 3, lines 37-54). Joskowicz 
further discloses (col. 3, lines 55-58) that M [g]iven the 
dimensions of the display screen, the task is to find the 
positions and dimensions of the rectangles for all the episodes 
in a clique such that they all fit on the screen and satisfy the 
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constraints." The spatial layout process consists of: (1) 
obtaining initial placements for the episodes in the clique, and 
(2) negotiating space requirements between them. The initial 
layout can be refined by the author by resizing or moving the 
corresponding rectangles (col. 3, lines 58-65). With an overlap 
on the screen, the depths of the overlapping episodes becomes 
important. Z-ordering defines the relative depths for the 
overlapping episodes as they appear on the display screen. 

The Z-list may be shuffled according to the following 
constraints. Each constraint is associated with a priority, with 
1 being the highest priority (col. 4, lines 17-22) and (col. 4, 
lines 23-31) wherein it is disclosed that u l. No episode shall be 
hidden completely behind another. 2. A button should always be 
fully visible. A button is a defined area within the episode 
that a user can select and thereby invoke a process linked to 
that area. 3. A portion of a text may not be hidden behind 
another episode. 4. An episode that takes up a larger area may 
be behind a smaller one." Conflicts among Z-order constraints 
are resolved according to their priorities: e.g., if a button and 
a text episode overlap, The button should still be fully visible 
(col. 4, lines 32-34) . 
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In the process according to the invention, the user is first 
given the opportunity to change or update the constraints 
(decision block 302 in figure 3). However, the user will most 
likely not change the constraints initially. Then, the system 
determines the layout (col. 4, lines 33-44). If the layout is 
not consistent, the Z-list can be modified by the author. For a 
given clique, we let forefront (c) denote an episode that is in 
the forefront of all of the episodes in the story, and we assign 
it the greatest depth from the point of viewing from front to 
back. An algorithm reorders the Z-list whenever an episode 
becomes the new forefront (col. 4, lines 52-56) . Thus, just by 
touching an episode on the screen, it becomes the current 
forefront (col. 4, lines 66 and 67). Although the positions of 
episodes are used as depth indices at the time of presentation, 
the positions of episodes are used only to determine the depth 
relationships between episodes of a clique (col. 5, lines 1-4). 

Joskowicz further discloses (col. 5, lines 11-15) that in the 
invention, there are two types of constraints, base and relative. 
Base relates to constraints concerning size (area) and relative 
relates to spacing rules specified by the author. In placement 
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on the screen, objects are spread on the screen by maximizing the 
sum of the distances between pairs (col, 6, lines 65-67). For a 
given clique, the goal is to automatically obtain a layout which 
is legal (the layout satisfies all the base constraints (size), 
Z-order constraints (depth) , and the relative spacing constraints 
(col. 6, lines 9-12). If a consistent layout is not found, the 
author may specify a new set of constraints and repeat the design 
process until a desirable design is found (col. 7, lines 13-16). 

With respect to resolution of Z-order constraints, 
initially, the Z-list is empty, and the objects are sorted by 
decreasing order of area; i.e., the larger area objects are 
displayed at a greater depth than smaller area objects so as not 
to occlude a smaller area object (col. 7, lines 24-31). To 
change the Z-order, the user chooses an object to bring it to the 
top of the display, by selecting the object with a pointing 
device, such as a mouse. The Z-list is then reordered (col. 7, 
lines 47-54) . 

The invention described so far relates to designing a screen 
layout for a given clique. For a series of cliques, we use a 
series of properties called constraint propagation. If an object 
is placed at a location (x,y) and has dimensions (w,h) the same 
object may be displayed along with another group after some time. 
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When this happens, rather than moving the object around, we may 
fix the location and dimensions and let the other objects move 
around the fixed object. This is what is meant by constraint 
propagation (col. 7, line 59 through col. 8, line 3). 

From the disclosure of Joskowicz, it is clear that the 
reference does not associate depth with at least one parameter of 
an object associated with an episode or thumbnail, but rather 
renders the episodes as a function of the size and aspect ratio 
of the episode itself, as well as the relative spacing between 
episodes, all as the presentation of a spatial layout. As 
Joskowicz is directed to providing a spatial layout where the 
episodes are properly spaced so the larger episodes do not 
occlude the viewing of smaller episodes, Joskowicz fails to teach 
or suggest that for each of the thumbnails (see figure 2) the 
depth is a function of at least one parameter of the object 
associated with the thumbnail, as recited in claim 1. 

Thus, even if we combined the teachings of Robertson and 
Joskowicz, as advanced by the examiner, the combined teachings of 
the references falls short of teaching or suggesting all of the 
limitations of claim 1. 
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In addition, we do not agree with the examiner's motivation 
for combining the teachings of the references. The reasoning 
provided by the examiner (answer, page 6) is not a motivation at 
*all, but rather is a statement that upon making the modification, 
the resultant structure will result in the claimed invention, 
instead of providing a line of reasoning as to why an artisan 
would have been motivated to combine the teachings of the 
references . 

We are not persuaded by the examiner's assertion (answer, 
page 18) that Mi]n contrast to Appellant [s ' ] argument Robertson 
and Joskowicz teach the depth as a function of parameter (such as 
priority or arrangement of episodes of the object associated with 
the thumbnail." As discussed, supra , in Joskowicz, priority 
refers to constraints relating to episodes not being completely 
hidden by another episode, and that buttons should not be hidden. 
This refers to the size, aspect ratio and relative spacing 
between the episodes themselves, and not to depth as a function 
of at least one parameter of the object associated with the 
episode or thumbnail. 

From all of the above, we find that the examiner has failed 
to establish a prima facie case of obviousness of claim 1. 
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Application No. 09/335,640 

Accordingly, the rejection of claim 1 under 3 5 U.S. C. § 103(a) is 
reversed. As each of the other independent claims, i.e., claims 
34, 61 and 62 contain the same or similar language, the rejection 
of independent claims 34, 61 and 62 is reversed, along with 
claims 2, 3, 5-9, 11, 12, 14, 16-22, 24, 33, 35-40, 42, 44, 46- 
51, 53, 54, 56, 58-60 and 63, which depend therefrom. 

We turn next to the rejection of claim 4 as being 
unpatentable under 35 U.S.C. § 103(a) as being unpatentable over 
Robertson , Joskowicz and Baldwin. We reverse the rejection of 
claim 4 as the examiner has failed to show how Baldwin makes up 
for the deficiencies of Robertson and Joskowicz. 



To summarize, the decision of the examiner to reject claims 
1-9, 11, 12, 14, 16-22, 24, 33-40, 42, 44, 46-51, 53, 54, 56 and 
58-63 under 35 U.S.C. § 103(a) is reversed. 



CONCLUSION 
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REVERSED 




Administrative Patent Judge 




LEE E. BARRETT 
Administrative Patent Judge 
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Administrative Patent Judge 
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